Sakkara documentation in SETUP for non-RHOAI clusters (#138)

Author: dgrove-oss

setup.RHOAI-v2.13/CLUSTER-SETUP.md

@@ -10,7 +10,12 @@ Create `default-priority`, `high-priority`, and `low-priority` priority classes:
 oc apply -f setup.RHOAI-v2.13/mlbatch-priorities.yaml
 ```
 
-## Coscheduler
+## Scheduler Plugins
+
+MLBatch utilizes Kubernetes Scheduler Plugins to ensure gang scheduling of
+multi-Pod workloads and to pack `Pods` onto `Nodes` to reduce GPU fragmentation.
+
+### Coscheduler
 
 Install Coscheduler v0.28.9 as a secondary scheduler and configure packing:
 ```sh
@@ -24,6 +29,8 @@ oc patch deployment -n scheduler-plugins --type=json --patch-file setup.RHOAI-v2
 oc patch deployment -n scheduler-plugins --type=json --patch-file setup.RHOAI-v2.13/coscheduler-priority-patch.yaml scheduler-plugins-scheduler
 ```
 
+
+
 ## Red Hat OpenShift AI
 
 Create the Red Hat OpenShift AI subscription:

setup.RHOAI-v2.16/CLUSTER-SETUP.md

@@ -10,7 +10,12 @@ Create `default-priority`, `high-priority`, and `low-priority` priority classes:
 oc apply -f setup.RHOAI-v2.16/mlbatch-priorities.yaml
 ```
 
-## Coscheduler
+## Scheduler Plugins
+
+MLBatch utilizes Kubernetes Scheduler Plugins to ensure gang scheduling of
+multi-Pod workloads and to pack `Pods` onto `Nodes` to reduce GPU fragmentation.
+
+### Coscheduler
 
 Install Coscheduler v0.28.9 as a secondary scheduler and configure packing:
 ```sh
@@ -24,6 +29,8 @@ oc patch deployment -n scheduler-plugins --type=json --patch-file setup.RHOAI-v2
 oc patch deployment -n scheduler-plugins --type=json --patch-file setup.RHOAI-v2.16/coscheduler-priority-patch.yaml scheduler-plugins-scheduler
 ```
 
+
+
 ## Red Hat OpenShift AI
 
 Create the Red Hat OpenShift AI subscription:

setup.RHOAI-v2.17/CLUSTER-SETUP.md

@@ -10,7 +10,12 @@ Create `default-priority`, `high-priority`, and `low-priority` priority classes:
 oc apply -f setup.RHOAI-v2.17/mlbatch-priorities.yaml
 ```
 
-## Coscheduler
+## Scheduler Plugins
+
+MLBatch utilizes Kubernetes Scheduler Plugins to ensure gang scheduling of
+multi-Pod workloads and to pack `Pods` onto `Nodes` to reduce GPU fragmentation.
+
+### Coscheduler
 
 Install Coscheduler v0.28.9 as a secondary scheduler and configure packing:
 ```sh
@@ -24,6 +29,8 @@ oc patch deployment -n scheduler-plugins --type=json --patch-file setup.RHOAI-v2
 oc patch deployment -n scheduler-plugins --type=json --patch-file setup.RHOAI-v2.17/coscheduler-priority-patch.yaml scheduler-plugins-scheduler
 ```
 
+
+
 ## Red Hat OpenShift AI
 
 Create the Red Hat OpenShift AI subscription:

setup.k8s/CLUSTER-SETUP.md

@@ -1,7 +1,7 @@
 # Cluster Setup
 
 The cluster setup installs and configures the following components:
-+ Coscheduler
++ Scheduler Plugins
 + Kubeflow Training Operator
 + KubeRay
 + Kueue
@@ -16,7 +16,13 @@ Create `default-priority`, `high-priority`, and `low-priority` priority classes:
 kubectl apply -f setup.k8s/mlbatch-priorities.yaml
 ```
 
-## Coscheduler
+## Scheduler Plugins
+
+MLBatch utilizes Kubernetes Scheduler Plugins to ensure gang scheduling of
+multi-Pod workloads and to pack `Pods` onto `Nodes` to reduce GPU fragmentation.
+Two options are described below: Coscheduler and Sakkara. You should pick and install one of them
+as a secondary scheduler for your cluster.
+### Coscheduler
 
 Install Coscheduler v0.28.9 as a secondary scheduler and configure packing:
 ```sh
@@ -30,6 +36,17 @@ kubectl patch deployment -n scheduler-plugins --type=json --patch-file setup.k8s
 kubectl patch deployment -n scheduler-plugins --type=json --patch-file setup.k8s/coscheduler-priority-patch.yaml scheduler-plugins-scheduler
 ```
 
+### Sakkara
+
+[Sakkara](https://github.com/atantawi/scheduler-plugins/tree/sakkara) is an experimental
+new scheduler plugin with advanced support for topology-aware scheduling.
+
+Install Sakkara as a secondary scheduler:
+```sh
+helm install sakkara-scheduler --namespace sakkara-scheduler --create-namespace mlbatch/sakkara-scheduler
+```
+Optionally, create a config map capturing your cluster's topology as described in the [Sakkara documentation](https://github.com/atantawi/sakkara-deploy/tree/main?tab=readme-ov-file#cluster-topology). This step is optional but recommended for production clusters. If the config map is not present Sakkara will default to a single-level hierarchy containing the Nodes of the cluster.
+
 ## Install Operators
 
 Create the mlbatch-system namespace
@@ -38,8 +55,14 @@ kubectl create namespace mlbatch-system
 ```
 
 Install the Kubeflow Training Operator
+
+If you are using Coscheduler do:
+```sh
+kubectl apply --server-side -k setup.k8s/training-operator/coscheduler
+```
+If you are using Sakkara do:
 ```sh
-kubectl apply --server-side -k setup.k8s/training-operator
+kubectl apply --server-side -k setup.k8s/training-operator/sakkara
 ```
 
 Install the KubeRay Operator
@@ -53,13 +76,19 @@ kubectl apply --server-side -k setup.k8s/kueue
 ```
 
 Install the AppWrapper Operator
+If you are using Coscheduler do:
 ```sh
-kubectl apply --server-side -k setup.k8s/appwrapper
+kubectl apply --server-side -k setup.k8s/appwrapper/coscheduler
 ```
+If you are using Sakkara do:
+```sh
+kubectl apply --server-side -k setup.k8s/appwrapper/sakkara
+```
+
 The provided configuration differs from the default configuration of the
 operators as follows:
 - Kubeflow Training Operator:
-  - `gang-scheduler-name` is set to `scheduler-plugins-scheduler`,
+  - `gang-scheduler-name` is set to either `scheduler-plugins-scheduler` or `sakkara-scheduler`,
 - Kueue:
   - `batch/job` integration is disabled,
   - `manageJobsWithoutQueueName` is enabled and configured via `managedJobsNamespaceSelector` to be
@@ -70,7 +99,7 @@ operators as follows:
   - `enableClusterQueueResources` metrics is enabled,
 - AppWrapper operator:
   - `userRBACAdmissionCheck` is disabled,
-  - `schedulerName` is set to `scheduler-plugins-scheduler`,
+  - `schedulerName` is set to `scheduler-plugins-scheduler` or `sakkara-scheduler`,
   - `queueName` is set to `default-queue`,
 - pod priorities, resource requests and limits have been adjusted.
 

setup.k8s/UNINSTALL.md

@@ -20,4 +20,8 @@ kubectl delete clusterrole mlbatch-edit
 # Coscheduler uninstall
 helm uninstall -n scheduler-plugins scheduler-plugins
 kubectl delete namespace scheduler-plugins
+
+# Sakkara uninstall
+helm uninstall -n sakkara-scheduler sakkara-scheduler
+kubectl delete namespace sakkara-scheduler
 ```

setup.k8s/appwrapper/kustomization.yaml setup.k8s/appwrapper/base/kustomization.yaml

@@ -17,6 +17,5 @@ images:
   newTag: v0.30.0
 
 patches:
-- path: config_patch.yaml
 - path: manager_resources_patch.yaml
 - path: remove_default_namespace.yaml

setup.k8s/appwrapper/manager_resources_patch.yaml setup.k8s/appwrapper/base/manager_resources_patch.yaml

@@ -0,0 +1,10 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+namespace: mlbatch-system
+
+resources:
+- ../base
+
+patches:
+patches:
+- path: config_patch.yaml

setup.k8s/appwrapper/remove_default_namespace.yaml setup.k8s/appwrapper/base/remove_default_namespace.yaml

@@ -0,0 +1,23 @@
+kind: ConfigMap
+apiVersion: v1
+metadata:
+  name: appwrapper-operator-config
+  namespace: appwrapper-system
+data:
+  config.yaml: |
+    appwrapper:
+      enableKueueIntegrations: true
+      kueueJobReconciller:
+        manageJobsWithoutQueueName: true
+        waitForPodsReady:
+          enable: false
+      defaultQueueName: default-queue
+      schedulerName: sakkara-scheduler
+      slackQueueName: slack-cluster-queue
+      userRBACAdmissionCheck: false
+    controllerManager:
+      health:
+        bindAddress: ":8081"
+      metrics:
+        bindAddress: "127.0.0.1:8080"
+      leaderElection: true

setup.k8s/appwrapper/config_patch.yaml setup.k8s/appwrapper/coscheduler/config_patch.yaml

@@ -0,0 +1,10 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+namespace: mlbatch-system
+
+resources:
+- ../base
+
+patches:
+patches:
+- path: config_patch.yaml

setup.k8s/appwrapper/coscheduler/kustomization.yaml

@@ -10,7 +10,6 @@ spec:
       - name:  training-operator
         args:
         - "--zap-log-level=2"
-        - "--gang-scheduler-name=scheduler-plugins-scheduler"
         resources:
           requests:
             cpu: 100m

setup.k8s/appwrapper/sakkara/config_patch.yaml

@@ -0,0 +1,15 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+namespace: mlbatch-system
+
+resources:
+- ../base
+
+patches:
+- target:
+    kind: Deployment
+    name: training-operator
+  patch: |
+    - op: add
+      path: /spec/template/spec/containers/0/args/-
+      value: "--gang-scheduler-name=scheduler-plugins-scheduler"

setup.k8s/appwrapper/sakkara/kustomization.yaml

@@ -0,0 +1,15 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+namespace: mlbatch-system
+
+resources:
+- ../base
+
+patches:
+- target:
+    kind: Deployment
+    name: training-operator
+  patch: |
+    - op: add
+      path: /spec/template/spec/containers/0/args/-
+      value: "--gang-scheduler-name=sakkara-scheduler"

setup.k8s/training-operator/kustomization.yaml setup.k8s/training-operator/base/kustomization.yaml

@@ -6,7 +6,7 @@ cluster roles, and priority classes.
 
 {{- else -}}
 The cluster setup installs and configures the following components:
-+ Coscheduler
++ Scheduler Plugins
 + Kubeflow Training Operator
 + KubeRay
 + Kueue
@@ -23,7 +23,15 @@ Create `default-priority`, `high-priority`, and `low-priority` priority classes:
 {{ .KUBECTL }} apply -f setup.{{ .VERSION }}/mlbatch-priorities.yaml
 ```
 
-## Coscheduler
+## Scheduler Plugins
+
+MLBatch utilizes Kubernetes Scheduler Plugins to ensure gang scheduling of
+multi-Pod workloads and to pack `Pods` onto `Nodes` to reduce GPU fragmentation.
+{{ if not .OPENSHIFT -}}
+Two options are described below: Coscheduler and Sakkara. You should pick and install one of them
+as a secondary scheduler for your cluster.
+{{- end }}
+### Coscheduler
 
 Install Coscheduler v0.28.9 as a secondary scheduler and configure packing:
 ```sh
@@ -37,6 +45,19 @@ Patch Coscheduler pod priorities:
 {{ .KUBECTL }} patch deployment -n scheduler-plugins --type=json --patch-file setup.{{ .VERSION }}/coscheduler-priority-patch.yaml scheduler-plugins-scheduler
 ```
 
+{{ if not .OPENSHIFT -}}
+### Sakkara
+
+[Sakkara](https://github.com/atantawi/scheduler-plugins/tree/sakkara) is an experimental
+new scheduler plugin with advanced support for topology-aware scheduling.
+
+Install Sakkara as a secondary scheduler:
+```sh
+helm install sakkara-scheduler --namespace sakkara-scheduler --create-namespace mlbatch/sakkara-scheduler
+```
+Optionally, create a config map capturing your cluster's topology as described in the [Sakkara documentation](https://github.com/atantawi/sakkara-deploy/tree/main?tab=readme-ov-file#cluster-topology). This step is optional but recommended for production clusters. If the config map is not present Sakkara will default to a single-level hierarchy containing the Nodes of the cluster.
+{{- end }}
+
 {{ if .OPENSHIFT -}}
 ## Red Hat OpenShift AI
 
@@ -115,8 +136,14 @@ Create the mlbatch-system namespace
 ```
 
 Install the Kubeflow Training Operator
+
+If you are using Coscheduler do:
+```sh
+{{ .KUBECTL }} apply --server-side -k setup.{{ .VERSION }}/training-operator/coscheduler
+```
+If you are using Sakkara do:
 ```sh
-{{ .KUBECTL }} apply --server-side -k setup.{{ .VERSION }}/training-operator
+{{ .KUBECTL }} apply --server-side -k setup.{{ .VERSION }}/training-operator/sakkara
 ```
 
 Install the KubeRay Operator
@@ -130,13 +157,19 @@ Install Kueue
 ```
 
 Install the AppWrapper Operator
+If you are using Coscheduler do:
 ```sh
-{{ .KUBECTL }} apply --server-side -k setup.{{ .VERSION }}/appwrapper
+{{ .KUBECTL }} apply --server-side -k setup.{{ .VERSION }}/appwrapper/coscheduler
 ```
+If you are using Sakkara do:
+```sh
+{{ .KUBECTL }} apply --server-side -k setup.{{ .VERSION }}/appwrapper/sakkara
+```
+
 The provided configuration differs from the default configuration of the
 operators as follows:
 - Kubeflow Training Operator:
-  - `gang-scheduler-name` is set to `scheduler-plugins-scheduler`,
+  - `gang-scheduler-name` is set to either `scheduler-plugins-scheduler` or `sakkara-scheduler`,
 - Kueue:
   - `batch/job` integration is disabled,
   - `manageJobsWithoutQueueName` is enabled and configured via `managedJobsNamespaceSelector` to be
@@ -149,7 +182,7 @@ operators as follows:
   - `enableClusterQueueResources` metrics is enabled,
 - AppWrapper operator:
   - `userRBACAdmissionCheck` is disabled,
-  - `schedulerName` is set to `scheduler-plugins-scheduler`,
+  - `schedulerName` is set to `scheduler-plugins-scheduler` or `sakkara-scheduler`,
   - `queueName` is set to `default-queue`,
 - pod priorities, resource requests and limits have been adjusted.