Operator to manage KAI-scheduler services

[enoodle]

To allow more fluid configuration changes and better support to running several schedulers we want to have a controller that will manage two CRD types:

1. KAI configuration that will describe the general parameters of the system with specific sections for each service
2. SchedulingShards that will describe how to partition the nodes in the cluster for different schedulers.

• Create basic config.kai.scheduler CRD
• Build an operator that will manage one config.kai.scheduler with hard coded name to setup just one the kai service in that namespace.
• Update the chart to use the new operator
• Add all the other services into KAI operator
  • Binder
  • Pod Grouper
  • Node Scale Adjuster
  • Queue Controller
  • Admission
• Create schedulingShard.kai.scheduler CRD
• Add to the operator a controller for SchedulingShard to better control the deployment of schedulers in the cluster.

[efenex]

I have just upgraded to v0.9.0 with this functionality included, and appear to be running into (potentially openshift-specific) issues. Mainly, only 3 services are deployed and running after a succesful Helm install:

NAME                            READY   STATUS    RESTARTS   AGE
binder-589f56df77-dnkv2         1/1     Running   0          16m
kai-operator-7d674c7ccd-vzfhw   1/1     Running   0          16m
scheduler-86d5b8c7f-djt7r       1/1     Running   0          16m

Looking at the deployments, replicas are requested but not spinning up. Detailed error message:

pods "pod-grouper-6b48bcff89-" is forbidden: unable to validate against
      any security context constraint: [provider "anyuid": Forbidden: not usable by
      user or serviceaccount, provider restricted-v2: .containers[0].runAsUser: Invalid value:
      10000: must be in the ranges: [1001030000, 1001039999], provider restricted:
      .containers[0].runAsUser: Invalid value: 10000: must be in the ranges: [1001030000,
      1001039999], provider "nonroot-v2": Forbidden: not usable by user or serviceaccount,
      provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid":
      Forbidden: not usable by user or serviceaccount, provider "logging-scc": Forbidden:
      not usable by user or serviceaccount, provider "machine-api-termination-handler":
      Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2":
      Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden:
      not usable by user or serviceaccount, provider "hostaccess": Forbidden: not
      usable by user or serviceaccount, provider "nfd-worker": Forbidden: not usable
      by user or serviceaccount, provider "nvidia-driver": Forbidden: not usable by
      user or serviceaccount, provider "nvidia-gpu-feature-discovery": Forbidden:
      not usable by user or serviceaccount, provider "nvidia-mig-manager": Forbidden:
      not usable by user or serviceaccount, provider "nvidia-node-status-exporter":
      Forbidden: not usable by user or serviceaccount, provider "nvidia-operator-validator":
      Forbidden: not usable by user or serviceaccount, provider "nvidia-dcgm-exporter":
      Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden:
      not usable by user or serviceaccount, provider "nvidia-dcgm": Forbidden: not
      usable by user or serviceaccount, provider "privileged": Forbidden: not usable
      by user or serviceaccount]

Note that this is a cluster-breaking issue due to the admission service no longer being served (!). I edited the configs.kai.scheduler and disabled admission and queueController services completely in order to remove the ValidatingWebhookConfiguration objects.

[efenex]

Just updating that I managed to workaround (temporarily) the above issue with the following yaml:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kai-admission-anyuid
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:openshift:scc:anyuid
subjects:
- kind: ServiceAccount
  name: admission
  namespace: kai-scheduler
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kai-pod-grouper-anyuid
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:openshift:scc:anyuid
subjects:
- kind: ServiceAccount
  name: pod-grouper
  namespace: kai-scheduler
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kai-podgroup-controller-anyuid
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:openshift:scc:anyuid
subjects:
- kind: ServiceAccount
  name: podgroup-controller
  namespace: kai-scheduler
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kai-queue-controller-anyuid
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:openshift:scc:anyuid
subjects:
- kind: ServiceAccount
  name: queue-controller
  namespace: kai-scheduler

(disclaimer: this allows the failing services to run as any user - absolutely not a secure solution)

After adding these permissions, I can get the deployments to start, but when scheduling a workload I end up with:

Warning  FailedCreatePodSandBox  1s (x3 over 17s)  kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to find runtime handler nvidia from runtime list map[crun:0xc000596a00 runc:0xc000596780]

I see the reservation pod uses the following snippet in the pod definition:

runtimeClassName: nvidia

I am reverting to an earlier version (0.5.x) to go back to a working configuration

[efenex]

Reverted to 0.5.3 and GPU workloads are scheduled again. Reviewing the YAML definition for the reservation pod does not show the runtimeClassName configuration in this version (but there may be other differences).

[enoodle]

@efenex Hello 👋

thank you for reporting it.

I don't see that we had any SCC / openshift specific configuration in earlier versions, but this is definitely something that we can add.
Could you open a dedicated ticket for that? It would have to include adding the correct SCCs, fixing the runtime class for reservation pods and also setting up automated tests with minishft or similar openshift-in-a-box solution.